Model assessment is a critical step in the process of predictive model machine learning techniques. Because the input data used to train a predictive model may include events that are relatively rare, oversampling of the input data is commonly used to pre-process the input data. The oversampled data is partitioned into training and validation datasets in which a training dataset is used to develop the predictive model, and the validation dataset is used to assess the trained model's performance. For model assessment, traditional assessments include a misclassification rate, an area under a receiver operating curve (ROC) (AUC), an F1 statistic, etc.
Many widely used predictive model assessment metrics are derived from a confusion matrix that is calculated based on an event decision threshold selected for the predictive model to indicate whether an event type has or has not occurred. For example, an event type may be a fraud event type, a device failure event type, etc. where occurrence of the event type is rare compared to occurrence of the non-event type such as a non-fraud event type, a device normal operating mode event type, etc. For illustration, the device may be a sensor, a computer, an industrial machine, a power transformer, an engine, an ATM machine, a pump, a compressor, etc.
There are primarily three limitations using this traditional approach. First, the dataset for model assessment is oversampled resulting in a very different distribution from the original input data that included certain types of events rarely. Second, the traditional assessments for traditional supervised learning models do not include objectives that include value/cost minimization based on misclassifications. Third, some traditional assessments do not determine a threshold probability value for prediction, which is critical to using a deployed predictive model.